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I. INTRODUCTION 


A. BACKGROUND 

mien Runy Payoff -from Office Computers." That was the 
Cover story for Fortune Magazine 26 May 1986. Why is this? The 
computer age was to bring increases in white-collar 
productivity as earlier generations of technological 
Buxcvatron did for other sectors, e.g. farming and 
manufacturing. So far, business investment of hundreds of 
billions of dollars in computer systems has failed to 
Significantly improve productivity. There has been little or 
MM nange in white-collar productivity since the sixties, 
where 85-90 percent of information technology is invested 
(Fortune 26 May 1986). Combined with the fact that 
"information technology is now the largest line item in 
Corporate America’s capital spending budget ... Companies need 
to develop better ways of measuring office productivity and 
the contribution that managers and other white-collar workers 


make to the bottom line." (Business Week, 26 November 1990) 


B. OBJECTIVES 

Evaluation cof ¡information systems © 1S necessary to 
effectively manage them as organizational resources. To do 
this an organization must be able to measure the costs and 


benefits related to its systems. This allows decisions on 


design and implementation to "be made in the same way as every 
other firm investment decision, i.e. based on its contribution 
to the success of the firm." (Marsden and Pingrey, 1988) 

Various evaluation methods have been used to assess 
information system effectiveness. The one of special interest 
for this thesis is user satisfaction. A number of artic NN 
IS literature cite user satisfaction as a surrogate for 
effectiveness (Ives, Olson and Baroudi, 1983; Pearson, 1977; 
Seward, 1975; Hamilton and Chervany, 1981). The desire for a 
surrogate stems from the fact that unlike transaction 
processing systems, data on IS use provides little information 
on the effectiveness of the white collar world. The issue here 
is whether opinions about a system such as user satisfaction, 
are the same thing as effectiveness of a system. MIS is a 
multidisciplinary field Spanning technical sciences, 
management science, psychology and other social sciences. 
Problems can arise when techniques, theories or results from 
a study in one field are used in another. Management science 
has had difficulty in establishing a link between satisfaction 
and productivity. This thesis will review the MIS literature 
that attempts to show a relationship between user satisfaction 
and system effectiveness. It will examine the development of 
this relationship in the MIS literature. Specifically, it will 
seek answers to the following questions: 


due ls user satisfaction a useful measuremen=w ot, 1. 
effectiveness? 


Do What are the various meanings of user satisfaction 
SH < 


E Does the IS literature now assume that measuring 
user satisfaction is equivalent to measuring 
effectiveness for IS, if so, does it reference 
previous literature? 

4. Are other methods used to measure effectiveness? 


What are they? Does the author give a reason for 
the Choice? 


C. SCOPE 

tess thesis > will investigate the measurement of 
information system effectiveness over the past 20 years. It 
examines what methods have been employed in the MIS 
literature. In particular, attention is focused on the use of 
user satisfaction as a surrogate for effectiveness by the MIS 


semmunity. Specific areas of interest include: 


I What method of measuring system effectiveness was 
employed? 
2. How user satisfaction was iniedi 


determined/defined. 


i9 The eol gua or OU the user satisfaction- 
effectiveness relationship and its current state. 


4, How and why user satisfaction is employed as a 
surrogate for effectiveness. 


a. Whether the employment of user satisfaction was 
JUSeltred e... COT rect... 
The data gathered will provide information as to the 


various methods of measuring IS effectiveness as well as 


2 on regarding the recent prevalence of user 


Satisfaction as the preferred method. 


D. METHODOLOGY 

A literature review will be undertaken to document the 
development IE the user Satisfaction-effectiveness 
relationship and the use of user satisfaction as a surrogate 
for IS effectiveness. It attempts to trace the origin of the 
relationship through to its current state. Attention will be 
focused on these questions: 

TM How was effectiveness measured? 


Zu Were other methods cónsidered and if so were there 
references cited? 


R If user satisfaction was chosen: 
a. Was the purpose to show user satisfaction as 
equivalent to effectiveness? or, 
b. Did the study assume equivalence and if so did 
it provide a reference? 

Defining and classifying what user satisfaction is and is 
not and what it can and cannot be used for was accomplished by 
drawing analogies from the sociological research by Denzin 
(1989) and Phillips (1971) focusing in particular TS 
descriptions and limitations of the survey technique. This 
parallel was felt justified based on the common psychological 
nature of the measure and the technique used for its 


employment, the survey, between sociological research and the 


user satisfaction measure of IS research. BaSically, it was 
determined that user satisfaction as an opinion, i.e., 
subjective measure should not be employed as an equivalent to 


effectiveness, an action. 


E. BENEFITS 

User satisfaction has aggressively been developed as the 
primary method of measuring IS effectiveness. MIS needs to 
know if this is a proper measurement of its work. What we know 
about information systems we know through the methods by which 
we observe them and the concepts by which we mentally model 
them. If the concepts are erroneous we are vulnerable to 


investing in systems that, in reality, are ineffective. 


II. BACKGROUND 


A. WHY MEASURE IS EFFECTIVENESS? 

"Information technology is now the largest line item in 
Corporate America's capital spending budget, accounting for 
nearly a third of current-dollar equipment outlays." (Business 
Week, 26 November 1990) The need to understand and measure 
information system effectiveness results from the fact that 
"U.S. business has spent hundreds of billions of dollar mEn 
them [computers], but white-collar productivity is no mikee 
than it was lin the late sixties.” (Fortune Lo May AA 

Organizations have invested in information technology to 
increase productivity, specifically white-collar productivity. 
"Economists calculate that about 85% of investment in 
information technology is made by the service [white-collar] 
sector. Yet non-manufacturing productivity has risen 
cumulatively just 1.9% over the past 10 years. 
‘Unfortunately,’ observes Morgan Stanley economist Stephen 
Roach, ‘the service sector has devoted far more effort to 
deploying computers than in determining the precise ways they 
can be used to enhance white-collar productivity’ 
Companies need to develop better ways of measuring office 


productivity and the contributions that managers and other 


white-collar workers make to the bottom line." (Business Week, 
26 November 1990) 

Clearly then, increasing the effectiveness of information 
systems 1S a primary objective of business organizations. To 
do so, they need to know what factors affect and contribute to 
system effectiveness. Bu is this understanding QE 
effectiveness which MIS research must provide so that 
iMiemeases in productivity and thus profits are realized, 
thereby justifying the current magnitude of investment in 
information systems by U.S. businesses. 

1. The Goal of MIS Research 

The goal of research is to increase the understanding 
of some relationship or situation. The goal of MIS research is 
to better describe and understand the LaGeers. ‘sand 
contributions of information systems to an organization. "MIS 
research is the systematic investigation of the development, 
operation, use and/or impact of an information (sub)system in 
an organizational environment." (Ives, Hamilton and Davis, 
1980). The goal of information systems is to increase the 
effectiveness of the organization, where effectiveness is the 
ability of performance to meet the goals and objectives set by 
the organization. The purpose of evaluation is to assess or 
appraise the value of an activity and measure its 


effectiveness (Arnovick and Gee, 1978). 


How MIS measures effectiveness is critical to the 
evaluation process and ultimately the overall understanding of 
information systems. 

2. Measuring Effectiveness 

Hamilton and Chervany £(1981) present two different 
views of system effectiveness. Each has its own meaning of 
System effectiveness and how it should be measured. In the 
goal-centered view "Effectiveness is determined by comparing 
performance to objectives." An example of this approach would 
be cost-benefit analysis. The second is the system-resource 
view where effectiveness is seen as "attainment of a normative 
state, e.g., standards for ‘good’ practices" rather than 
establishing specific task objectives. 

The complementary nature of these views is similar to 
that of summative and formative evaluation approaches. The 
focus of summative evaluation is the accomplishment of 
objectives. These are tangible factors capable of objective 
measurement. Formative evaluation is assessment of quality and 
consist of intangible factors measured subjectively. "Both 
formative and summative evaluation approaches are typically 
used in providing evaluative information on system 
effectiveness." (Hamilton and Chervany, 1981). 

It is importane to point out the significance of the 
dual nature of the evaluation process and the contributions 


and limitations of each approach. 


a. Goal-Centered View 
Basically the  goal-centered view of system 
effectiveness focuses on the efficiency of the system. It 
assesses system performance with respect to established system 
and/or organizational objectives. Quantifiable (tangible) 
benefits provide objective measures of performance. Like the 
Summative evaluation approach, the goal-centered view provides 
"information on the systems outcomes, or ends, to support 
decisions to continue, adopt or terminate the system." 
(Hamilton and Chervany, 1981). That is, the objectives set by 
the organization are the critical factors against which the 
system success or failure is judged. 
b. System-Resource View 
The system-resource view uses subjective measures 
of intangible benefits to assess the effectiveness of the 
System. Like the formative evaluation approach, the system- 
resource view provides supplementary/explanatory information 


about the system (Hamilton and Chervany, 1981). 


B. THE REASON FOR THIS STUDY 

Hamilton and Chervany’s (1981) dual view of system 
effectiveness identifies these two system evaluation 
approaches as separate yet complimentary. This complimentary 
relationship is analogous to the one between summative and 
formative evaluation approaches where information provided by 


the formative evaluation helps “improve the means and process 


to accomplish objectives and interpretation of summative 
evaluation results." They further explain (LIM ns 
objective measures are provided by the "formal evaluation 
approaches" while "perceptions of system effectiveness" are 
gained by the informal ones which are "necessary and helpful 
for calibrating the credibility of information HORNOS 
evaluation. u 

Assessing the accomplishment of official goals and 
objectives and thus the overall effectiveness, or success, of 
the system is the function of the formal evaluation 
approaches, 1.e., goal-centered view, summative evaluation, 
and objective measures. The informal approaches; the system- 
resource view, formative evaluation, and subjective measures 
provide supplementary information, recognizing that "systems 
have other functions and have other consequences besides 
accomplishment of official objectives" (Hamilton and Chervany, 
T99 1E 

The message here is that the system-resource view, 
formative evaluation and/or subjective measures are informal 
approaches and are at best supplementary measures to "explain" 
or "help ... and aid." The point being that the goal-centered 
view, summative evaluation and objective measures are formal 
evaluation approaches and are the primary means of determining 
the success or failure of the system, to be supplemented by 


their "informal scouneerpares. 
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Relevant to the current study is the subjective measure, 
user Satisfaction, which is a perception of system 
effectiveness. Although Hamilton and Chervany suggest that 
"The user attitude survey [user satisfaction] is a formal 
approach to quantify subjective perceptions of system 
effectiveness." this thesis contends that user satisfaction is 
none the less just a supplementary measure of system 
effectiveness. 

The hazard in employing user satisfaction as a prime means 
of evaluating the effectiveness of information systems, to the 
exclusion of assessing costs and benefits, is that it is an 
opinion-based technique. An example of the hazard is 
illustrated by international surveys of students’ opinions of 
their math competency. In such surveys, U.S. high school 
Students rate themselves among the worlds best. In actual 
testing of math skills, U.S. high school students are among 
the world’s worst. The opinion is not the same as the skill. 
Similarly, users of information systems may have numerous 
motives to claim satisfaction with these systems. Yet, these 
opinion reports are not, conceptually, the same thing as an 
assessment of a system’s productivity in terms of inputs and 
outputs or its quality in terms of better management 
decisions. Indeed, in two studies where both effectiveness and 
Satisfaction were measured, the findings were contradictory, 
Gallupe and DeSanctis (1988), and Aldag and Power (1986). It 


is the distinction between primary and supplementary measures 


P 


that will serve as criteria for classifying articles reviewed 
in this thesis. The purpose will be to trace the genesis and 
evolution by which user satisfaction crowded out cost-benefit 


analysis. 
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III. METHODOLOGY 


A. SELECTION OF SYSTEM EFFECTIVENESS STUDIES 
1. Selection of Articles for Study 

Dus concept -of -this study has. as its impetus, 
questions raised in a previous thesis on information system 
effectiveness by  Mustofa (1989). Therefore, it seemed 
reasonable to begin article selection with those identified as 
dealing with systems effectiveness in the preceding study. To 
these were added articles published after 1988, the last year 
included in the Mustofa study. These new articles were 
collected by the same means, which will be briefly discussed. 
(The reader may refer to Chapter III Methodology, Mustofa 
(1989) for a detailed account of the search and selection 
process.) 

A consensus of journal names from three studies 
reporting on articles published in the MIS field was selected. 
Nue starting point was 1970 and continued through 1988. A 
DIALOG search from the INSPEC database was conducted by the 
Dudley Knox Library at the Naval Postgraduate School. The 
bibliographic search was on the following key: (Productivity 
or Effectiveness) and (Measures or Measurement or Assess or 
Assessment) and Information System. This automated search 


turned up a limited number of useful articles, fewer than 25. 


55 


Therefore, a manual search was conducted examining the 
periodicals on the consensus list between the years 1970 and 
1988. Indices or every issue of periodicals without an index, 
were examined for articles possibly dealing with information 
SyStem effectiveness. This search turned up 118 articles as 
well as the reason for the small number found during the on- 
line search, the limited number of key word synonyms for 
effectiveness and productivity. Additionally, articles were 
obtained from friends and colleagues aware of the situation or 
involved in related work as well as those published subsequent 
LOMLS SS UB to JU D 
2. Methods of Evaluating System Effectiveness 
a. Knowledge 

Knowledge accumulation is an important aspect in 
"the relationship between actions and outcomes" (Goldstein et 
al., 1990). Various methods of knowledge accumulation have 
been proposed and summarized by Goldstein et al. (1990): Weick 
(1979) believes that "individuals make sense of events 
retrospectively," that is, they assimilate information about 
the event(s) and then explain them retrospectively to 
understand them. Salancik and Porac (1986) have identified a 
three step process; gather data to understand cause and effect 
relationships, build a knowledge base of these relationships, 
and use them to come to generalized statements. Isenberg 


(1986) identified "a cycle of reasoning and action” using 
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existing knowledge identify any new information required and 
then act and update through feedback. Lord and Foti (1986) 
describe a knowledge representation scheme classifying 
different environmental elements which are used as prototypes 
and organized hierarchically for decision making. 

The amount of knowledge accumulated is linked to 
job performance. Experts were found to have not only more 
information available to them but it was also more useable 
i.e., they had more and better descripters, and more inter- 
See@egery information (Lurigio and Carroll, 1985). Goldstein et 
al. (1990) used a knowledge classification scheme of 
declarative, procedural and causal knowledge. "Declarative 
knowledge consists of facts and definitional descriptions of 
categories ... Procedural knowledge is the heuristics and 
rules that guide a person's actions ... Causal knowledge 
consists of the description of the existence, strength and the 
direction of the relationships among categories and 
strategies." 

b. Cost-Benefit-Analysis 

"A cost-benefit-analysis (CBA) is ECCE. a 
comparison between two states. Proposed new system costs and 
benefits are usually compared with those of current systems 
whether they be manual, partly computerized or fully 
computerized." (Lincoln, 1986). There is usually little 


difficulty identifying costs of the systems as these are 


leo 


usually well documented. However, identification of benefits 
and subsequently assigning a value to them has proven to be 
difficult. Basically there are two types of benefits, tangible 
and intangible. Tangible benefits are relatively easy to 
quantify and assign a value. Intangibles, on the other hand, 
have proven to be hard to quantify and when values were 
assigned they have been viewed skeptically. Often they are not 
even considered in the problem. There are different 
philosophies to the valuation of intangible benefits. Matlin 
(1979) argues for "placing dollar amounts on all intangible 
benefits" stating that "Difficulty in finding a value NE 
an appropriate reason for avoiding the subject. Tt TSS 
important to derive techniques for valuing benefits of 
information system investments. Even an approximation of the 
values will provide managers an opportunity to make informed 
decisions about future investments. Without these values 
decisions may be made without appropriate information." 
(Matlin; 1979) 

Another approach is made by Hall and Lincoln 
(1976). They consider only quantifiable benefits, estimate 
them conservatively and check "the financial results for 
accuracy and plausibility" reasoning that senior management 
has been made skeptical of CBA by "many examples of quick, 
glib, cost justificacion exercises. 

The value obtained is then used in a comparison 


between pre and post system introduction, but requires a 
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stable environment or a control group in place to be a valid 
measure. Alternatively, the value may be compared against 
another system, which however, requires "a reconstruction of 
the state of affairs assuming either an alternative system is 
in place or the activity was not undertaken." (Lincoln, 1976). 
Determining the basis for comparison i.e., previous system, 
alternative technology or manual system and the level of 
Eu Ice 1S complicated, difficult and uncertain. Senior 
executives often require initial comparison to a manual 
system. Its advantages include zero-based comparison and 
familiarity due to a large database of similar applications 
over many years. | 

CBA has been criticized because of the difficulty 
and subjectivity in identifying and/or evaluating the 
intangible benefits of unstructured, ad hoc decisions and 
obtaining determinable data supporting system success but 
munecorded by an organization (Ives et al., 1983). 

C. Value Analysis 

Value analysis is similar to user satisfaction upon 
which it is based but is preferred in evaluating DSS (Money et 
al., 1988). Value analysis is employed in evaluating DSS and 
begins with the premise that "information is data of value in 
decision making" (Yovits and Foulk, 1985). They also suggest 
But information gives rise to observable effects through the 


decision making process and information feedback exists so 
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that the decision maker (DM) will adjust his assessment of the 
decision situation for later decisions” (Yovits and sami 
1985). Money et al. (1988) using the method of com EM 
measurement convert subjective judgement of the value of 
intangible benefits into scores for analysis. 
d. Profit Maximization 

A key objective of an organization is profit 
maxim@zation. Information systems design should focus on 
achieving that goal. Marsden and Pingrey (1988) argue that 
decisions regarding IS be made as any other investment 
decision, "its contribution to the success of the firm une 
develop a goal drive mathematical process model which seeks 
the goal of profit maximization. Using values of revenue 
enhancement or cost reduction of various information products 
and costs of the necessary man-machine inputs of those 
products, they maximize the difference. Citing numerous 
references in the economics and IS fields to support their 
technique, Cron and Sobol (1983) use multiple documented 
financial performance measures; pretax return on assets, 
return on net worth, pretax profits as a percent of sales and 
average five-year growth as indicators of performance, i.e., 
effectiveness. 

e. Use 
Use of a computer system is expected to improve 


performance of its users (Lucas, 1974). Therefore the 
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relationship between use of a system and performance 
(effectiveness) 1S investigated. Effective systems are assumed 
to attain higher use than less effective ones. However, this 
Requires that use of the system is optional as for a DSS. With 
mandatory use of the system the relationship between use and 
effectiveness fails. Criticism of use as a measurement for 
effectiveness comes from the fact that it measures only the 
SUE to the system, where as the desired value 
(performance/effectiveness) concerns the output of the system. 
f. Productivity 

Productivity as a measure of system effectiveness 
is a before and after comparison of some performance 
objective(s). There are various levels of sophistication in 
defining these objectives, from measuring simple letter 
production using a word processing system (Kapsales, 1986) to 
sophisticated regression models of multiple variables 
(Englander and Englander, 1985). 

g. User Satisfaction 

User satisfaction is a measure of the users 
Subjective evaluation of factors concerning an information 
System's utility. These factors originate from work on 
organizational behavior by Cyert and March. They argue that 
the users "need for information" is either reinforced or 
Frustrated by the intormat Lon system, the success 


(effectiveness) of which is the users sense of satisfaction. 
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Most user satisfaction instruments use the semantic 
differential technique developed by Osgood, Sucima 
Tannenbaum, e.g., the Shultz-Slevin questionnaire. However, 
"Most of those that existed prior to an empirically "emi ed 
methodology developed by Pearson (1977) have been heavily 
criticized" (Evans et al., 1988). Bailey and Pearson NE 
developed a tool from Pearson's 1977 dissertation, III 
factors from "a review of 22 studies of the computer/user 
interface." The Bailey and Pearson tool is based on Wanous and 
Lawlers model defining satisfaction as "the sum of the user’s 


weighted reactions to a set of factors, 


DS N. 
i= 


where 


Rij The reaction to factor J by individu TEER 


Wij The importance of factor j to individo TEE 


This model suggests that user satisfaction is the sum of ones 


positive and negative reactions to a set of factors." (Bailey 
and Pearson, 1983). Bailey and Pearson also utilize the 


semantic differential technique of Osgood et al. 

Ives et al. (1983) reduced the Pearson tool to "a 
‘short form’ of the instrument for use when time is limited 
and/or only an overall assessment of UIS [user information 
satisfaction] (rather that analysis of particular symptoms of 


a problem) is adequate." 
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h. Multiple Criteria 
The general intent of multiple criteria studies is 
to more thoroughly “examine the Significance of various 
EuSoPsartion system characteristics" (Dickson et al., 1977) for 
"an easy-to-use instrument for improving the effectiveness of 
wo r mation systems” (Evans et al., 1988). Proponents of the 
aw G criteria approach conclude that "Information systems 
Enc product of multiple factors" (Mansour and Watson, 1980). 
In general, some combination of system performance, goal 
attainment and user criteria is employed. The intent being to 
more fully describe and define the IS environment, thereby 
more accurately evaluating the IS effectiveness (Mansour and 
Wwson, 1980; Chandler, 1982; Evans et al., 1988). Chandler 
suggests information systems be evaluated from two 
perspectives, one being the computer system domain the other 
the user domain, each having its own goals and measures. Evans 
et al. conclude that by applying different methodologies the 
various results "can work separately and in harmony with each 

Sener” (Evans et al., 1988). 

3. Evolution of the User Satisfaction Tool 

Cyert and March (1963) are credited by Bailey and 
Pearson (1983), and Ives, Olson and Baroudi (1983) with being 
the source of the user information sabttistaction (UES) Concept. 
Basically, they argue that “when an organization discovers a 


solution to a problem by searching in a particular way, it 
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will be more likely to search in that way in future problems 
of the same type; when it fails to III 
searching in a particular way, it will be less likely to 
search in that way in future problems of the same type." 
Drawing inference from this to information systems, 1t emer 
that the ability of the system to meet the needs of the user 
either reinforces or frustrates the users’ sense of 
satisfaction with the system. Therefore, it is suggested that 
"Satisfaction of users with their information system qe M 
potentially measurable, and generally acceptable, surrogate 
for utility in decision making" (Ives, Olson and Baraona 
VOGO 
User satisfaction is a subjective measure of system 
success. It relies on the users perceived value of the 
information system’s effectiveness to be a surrogate measure 
for objective measures deemed to difficult to obtain. 
a. Early User Satisfaction Measures 
The early measures of user satisfaction, those 
through 1977, had varied terminology for the concept of user 
Satisfaction, for example: "appreciation" (Swanson, 1974), 
"user attitudes" (Lucas, 1974), "user perceptions" (Gallagher, 
1974; Schewe, 1976), "perceived success" (Edstrom, 1977), 
"user evaluation" (Dickson, Senn and  Chervany, 19 7999 
Generally these measures were uniquely "designed" for the 


particular study that employed it. Often the definition and 
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E D T were not described in the article, 
see Appendix E, Table VII. 
b. The Shultz-Slevin Questionnaire 

The Shultz-Slevin Questionnaire is based on the 
Likert and semantic differential techniques. The -Likert 
instrument "consists of a number of statements to which the 
respondent indicates how strongly he agrees or disagrees ona 
five-point or seven-point scale" (Shultz and Slevin, 1975). 
The semantic differential is a scale that "consists of the 
Statement of a concept to which an individual responds by 
checking a set of bipolar adjectives; good-bad." The Shultz- 
Slevin Questionnaire became the prevalent method in the 
sueu@r1es reviewed from 1978 to 1983. The term user satisfaction 
became more common, although each study generally still had 
its own definition of "user satisfaction." However, the 
construct of the measure began to standardize around the 
Shultz-Slevin Questionnaire, see Appendix E, Table VII. 

c. The Pearson Methodology and the Bailey and Pearson 

Instrument 

Examining Table VII with respect to studies 
published after 1983 shows that either the Pearson Methodology 
or the Bailey and Pearson Instrument was almost the exclusive 
measure of user satisfaction when evaluating information 


system effectiveness. The Bailey and Pearson Instrument is 
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described in section (2), Methods Of Evaluating me P 
Effectiveness. 
Evans et al. (1988) sum up the evolution of user 
Satisfaction this way: 
Instruments to measure user satisfaction are scarce. Most 
of those that existed prior to an empirically derived 
methodology developed by Pearson (Pearson, 1977) have been 
heavily criticized (Ives, Olson and Baroudi, MIA 
Pearson’s Methodology, which has received wide acceptance, 
uses a semantic differential technique to rate 39 factors 
which cause satisfaction or dissatisfaction WWE 
information systems. The 39 factors were identified based 
on extensive literature review and expert opinion. (Evans 
et al., 1988). 
4. Selecting Articles Dealing With User Satisfaction 
The compilation of ‘articles for this study obtained 
from the search found 96 as possibly dealing with user 
Satisfaction or system effectiveness. These then were the 
Studies initially reviewed by this study and are listed in 
Appendix A. 
a. User Satisfaction Studies 
Reading the 96 articles in Appendix A yielded 40 
articles dealing with user satisfaction or one of its aliases 
(as described in the preceding section). The selected studies 
employed user satisfaction or promoted its’ employment as a 
measurement tool for some aspect of information system 
effectiveness. These included empirical studies on system 
effectiveness and methodological prescriptions for the 


employment of user Satisfaction in measuring system 


effectiveness. These studies are listed in Appendix B. 
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b. Alternative Measures of Effectiveness 
Appendix C lists the 19 studies included in enis 
section. These alternative measures of effectiveness are 
described in the preceding section. 
c. Miscellaneous Literature 
Appendix F contains the 29 remaining studies 
reviewed. Included here are: literature reviews, essays, 
puescriptive methodologies, frameworks and tools; and 10 
articles not specifically dealing with effectiveness 
measurement or were too unclear as to method or objective to 


evaluate properly. 


B. REVIEWING THE USER SATISFACTION STUDIES 
The 40 user satisfaction studies in Appendix B were 


evaluated with respect to the following criteria: 


es The conventional wisdom assertion (CWA). 

D Was the Justification Supported? 

E References rejecting alternative measures of 
effectiveness. 


1. The Conventional Wisdom Assertion (CWA) 

The idea of documenting the CWA occurred after the 
initial review of all articles. The impetus came from the 
following observations. One, that there were articles which 
directly claimed that user satisfaction was the generally 


accepted or prevalent method for measuring IS effectiveness. 
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Two, that often the only evaluation measure mentioned or 
considered was user satisfaction, 1.e., possible alternative 
measures of IS effectiveness did not receive mention let alone 
consideration. 

Determining which studies made the CWA category was 
then necessary. There were varying degrees of subtly 
encountered when trying to apply the CWA classification, from 
merely implied to outright claims of prevalence or general 
acceptance. The outright claims were relatively easy to 
identify, as most made the outright claim in the Abstract or 
Introduction. Some examples are: 

The measurement of how satisfied a user is with his or her 
information system (user information system or UIS) has 
become a pervasive measure of the success or effectiveness 
of an information system. (Baroudi and Orlikowski, 1988). 
To measure system success, researchers have mainly relied 
on elther of two proxies: user satisfaction or system use. 


(Barki and Hug 19955) 


User information satisfaction is the preferred indicator 
of system success ... (Tait and Vessey, 1988) 


One of the most widely used methods of evaluating system 
effectiveness is the user satisfaction survey." ^"... our 
rather wholesale acceptance of these [user satisfaction] 
evaluation tools. (Hufnagel, 1990) 


The primary consideration in evaluating Management 
Information Systems (MIS) effectiveness is that it ‘must 
be evaluated on the criteria of importance to the users'. 
(Lucas, 199.3) 

user satisfaction with MIS ... provides the most useful 
assessment of system effectiveness. (Hamilton and 
Chervany, 1981) 

Table I of Appendix E contains those studies that were 


identified as having made the CWA. Appendix D, Annotated 
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Mescription of - User Satisfaction Articles, contains the 
selected documentation evidence used for assigning the CWA 
eleassiticatlion to a Study. Included in the description are 
quotations and page references. For outright claims that user 
satisfaction is the prevalent or accepted method of measuring 
system effectiveness, this documentation serves as complete 
evidence for the CWA classification decision. 

Those that only implied the CWA were considerably more 
earricult to notice and especially difficult to document. This 
may seem contradictory to the notion of conventional wisdom. 
Maver. when taken to the extreme, conventional wisdom is an 
assumption by its user and, m such, is easily overlooked and 
Prerrcult to pinpoint by an outside party searching for 
specific documentation. Additionally, the purpose of some 
studies was to further develop or test a particular user 
satisfaction tool. Generally, for these cases 1t was often 
necessary to examine the Introduction, Methodology (for 
description of the measurement tool), and Conclusions sections 
to determine the extent of an author’s assumption regarding 
his intended and actual employment of the tool as a measure of 
effectiveness, with respect to the CWA. 

Mme) Sannotcated “description Jof! User  Satzsfaction 
Articles, Appendix D, contains relevant quotations concerning 
the CWA. It should be noted that the quotations from some 
studies do not clearly indicate a conventional wisdom 


Statement. However, they do at least suggest an attitude of 
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acceptance of the measure, which, when combined with other 
statements in the article, provided enough circumstantial 
evidence for it to be included in the CWA category. The 
circumstantial evidence which lead to the CWA judgement was 
mainly obtained from reading the Introduction, Methodology and 
Conclusions sections with respect to intent, actual employment 
and information provided about the system by the user 
Satisfaction tool. The following are some examples: 
In information systems(IS) research, user satisfaction (or 
user information satisfaction) is considered to be a key 
construct for assessing system performance. (Barki, 1990). 
While acknowledging the importance of economic analysis of 
MIS value, researchers respond to the shifting emphasis 
form efficiency to effectiveness by focusing either on MIS 
usage or user perceived effectiveness. (Srinivasan, 1985) 
2. Justification Supported 
Justification Supported was the classification given 
to studies which properly applied user satisfaction as an 
effectiveness measure. When determining whether the employment 
of user satisfaction was justified the criterion of this study 
was the degree to which the tool was to measure effectiveness. 
Was it as a "relative" measure or as an "absolute" measure of 
system effectiveness? In other words, a "better" or "more” 
effective system rather than that the system "was" or "was 
not” successful. The choice of relative or absolute measure 


was selected as best describing the myriad of approaches to 


employment user satisfaction has in the literature, from a 


uo 


Single item (factor) measure of early works, to multi-factor 
measures employed in a triangulation method. 

This study used as its basis for judgement, arguments 
from Denzin (1989) and Phillips (1971). These were chosen for 
their information concerning the survey technique, its use and 
I wstatgons. 

Denzin presents the theory of symbolic interactionism 
as the perspective from which to analyze research methods. 
Briefly summarized the concept of symbolic interactionism is: 

If human behavior is observable at two levels - the 
symbolic and the interactional - then central to 
understanding such behavior is the range and variety of 
symbols and symbolic meaning shared; communicated and 
Mamipulated by interacting selves in social situations. 
Society contributes two essential elements that reflect 
directly on concrete interactions: (1) the symbols; or 
various languages provided and communicated through the 
Socialization process, and (2) the concrete behavioral 
settings in which the interaction occurs. (Denzin, 1989) 

With respect to this study, the symbolic level is 
interpreted as the opinion expressed by the user regarding his 
or her satisfaction with the system and the interactional 
level is that of the observed or measured behavior, 
effectiveness or performance, of the user himself. Regarding 
the survey technique, Denzin states that " Users of the survey 
Seldom combine symbols with interaction ...” (Denzin, 1989). 

Phillips, when discussing interviews and 
questionnaires states that: 

... inferences about nonverbal behavior which are based on 
nonverbal responses are more subject to distortion and 


errors than are inferences about future or past nonverbal 
behavior which are based on the observation of such 
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behavior. That is, it is easier to 'lie with words EM 
te ‘lie with sacts Seen l lies, UM 


Here again the author questions the ability of the 
Survey technique to draw inference to nonverbal behavior, 
i.e., action. Furthermore, only some of the user satis: o EEE 
authors, Table VIII, have addressed questions of measurement 
validity and reliability. However, one particular type 
proposed by Phillips that seems pertinent has not, convergent- 
descriminant validity. This is the ability to inda Fans 
measures by showing that they "correlate poorly with some 
criterion but also by showing not only that they corr reike 
highly with some conceptually simpler variable, such as the 
tendency to respond true or in a socially desirable manner at 
all- -times "PF MAPS LS 

The basic premise in the current study is drawn from 
arguments by Denzin and Phillips, that is, that users 
responses on a user satisfaction survey regarding information 
system effectiveness (opinion) are not the same as measuring 
performance as a result of an information system. This study 
makes a differentiation between the relative measure, 
employment as a descriptor or supplementary measure of the 
system and the absolute measure, success or failure of the 
system as its means of classifing opinion (relative) and 
action (absolute). The delineation was made by drawing an 
anology to the opinion-performance arguements of Denzin and 


Phillips. Relative measures were viewed as anologous to 
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opinion, i.e. they can enhance or supplement objective or 
primary measures but cannot stand on their own. Absolute 
measures, on the other hand, have the necessary and sufficient 
conditions to stand on their own. The relative measures were 
designated Justification Supported while the absolute were 
pot. 

Deciding if a particular study's employment of user 
satisfaction was justified, this study first looked at whether 
the article being reviewed employed a multiple-criteria 
method, i.e., user satisfaction was supplementing information 
from another measurement technique. This ancillary use was 
determined to be a justified use of the user satisfaction tool 
and classified as a relative measurement thus was a case of 
Justification Supported. There was considerable latitude in 
applying the multi-criteria test. This broad interpretation 
was because this author was not skilled in determining the 
validity of the various research methods encountered and 
wished to be as conservative as possible in assigning 
restrictions to the method, thereby providing the researchers 
the widest possible latitude in their employment of the user 
Satisfaction tool. 

Studies which used user satisfaction as the sole means 
of measuring the system effectiveness constituted the other 
major category. It was necessary to determine the use of the 
user satisfaction  tool's value, was it for relative 


qualification or judgement of the system or was it for an 
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absolute quantification or description of the system’s success 
or failure. This was a particularly difficult assessment to 
make. As with the CWA, some authors stated directly how they 
intended to employ the tool, while others had conflicting 
statements in their articles which required analyzing the 
intent and conclusions drawn from the measure's employment. 
The following are examples of the basic Justification 
Supported categories: 
Justification Supported: Yes, relative measure. 
concerned only with identifying difference in the 
affective response patterns of the two groups of 
performers (as opposed to arriving at a 'true' measure of 
value. (Hufnagel, 1990) 
Justification Supported: No, absolute measure. 
The measurement of how satisfied a user is with his or her 
information system (user information satisfaction or UIS) 
has become a pervasive measure of the success or 
effectiveness of an information system. (Baroudi and 
Orlikowski, 1988) 
Justification Supported: Yes, multiple criteria. 
The purpose of this study was to apply four methodologies 
to help understand and measure current information systems 
The objective is to recommend an easy to use 
instrument for improving the effectiveness of information 
systems. (Evans et al., 1988) 
Table I of Appendix D contains the determination data 
on Justification Supported. Additionally it contains thek -ne 
number where relevant information leading to the judgement is 


located. Appendix D contains selected quotations which either 


provide direct evidence or substantial circumstantial evidence 


S 


documenting the study’s documentation Justification Supported 
decision. 

Another category of data in this thesis was whether 
there was a specific reference to rejection of an alternative 
measure(s) of system effectiveness, see Table V, Appendix E. 
All that was required was for a Study to include somewhere in 
its discussion and in its list of references an alternative 
measure. The point being, that there was an attempt made at 
selecting an appropriate method EO measure system 


Spece) veness. 
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IV. FINDINGS 


Appendix D contains an annotated description of each of 
the 40 user satisfaction studies evaluated here. Appendix E 
contains the detailed tables of findings for each aspect of 
this study. They are listed in chronological order by date of 
Publicat ione 

Studies in Appendix B employed a form of user satisfaction 
measure to evaluate the effectiveness of an information 
system. 

In Appendix C, the studies used an alternative measurement 


tool to evaluate system effectiveness. 


A. CONVENTIONAL WISDOM ASSERTION 

The first question for which an answer was sought was what 
number of articles assume user satisfaction is currently the 
conventional wisdom or prevalent method employed by the IS 
community when measuring the effectiveness of an information 
system. 

Table I of Appendix E condenses out of Appendix B the 
primary references cited by each author for choosing user 
Satisfaction as a surrogate measure of system effectiveness 
and for those with none listed the conventional wisdom 
assertion (CWA) was assigned. Table II shows that the CWA is 


the only reference for 45% of the MIS studies that employ user 


34 


satisfaction aS a measurement tool. This implies that MIS 
researchers presuppose not only that user satisfaction is the 
preferred method of measuring MIS effectiveness but that it 
has also been proven and accepted as such by the MIS 
Semmunity. This 1s certainly not the case as shown by Appendix 
ten lists 19 studies that use alternative methods of 
measuring an information system’s effectiveness. Additionally, 
it contains several references in MIS literature rejecting 
user satisfaction as an absolute or single value measurement 
of a system’s effectiveness (Ahituv, 1980; Marsden and 
Pingrey, Teo; Hufnagel, T990) on theoretical Or 
methodological grounds. 

Additional distillation of the data in Table I at years 
1977 (Pearson) and 1983 (Bailey and Pearson) provides further 
insight into the genesis and evolution of the CWA. In 1977 
Pearson's dissertation developed a tool for measuring user 
EEUNErHction. In 1983 Balley and Pearson further refined the 
tool in a published article (Bailey and Pearson, 1983). Evans 
et al. (1988) state that "Instruments to measure user 
satisfaction are scarce." Most of those that existed prior to 
an empirically derived methodology developed by Pearson have 
been heavily criticized. Now, examining the percentage of the 
CWA in studies from 1977 forward and 1983 forward it appears 
that the emergence of the CWA has it's origins with Pearson's 
1977 dissertation and it's widespread practice with the 


publishing of the 1983 Bailey and Pearson study. There is also 
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an increase in the number of empirical studies evaluating IS 
(Alavisét alae oes ie 

Studies with a CWA of user satisfaction have grasped this 
tool as the an easy and reliable way to measure system 
effectiveness. Often they have ignored the limitations stated 
by Bailey and Pearson concerning its use. Bailey and Pearson 
propose that it be used for "relative" evaluation, i.e. 
comparison of and between systems. Many authors making the CWA 
imply and draw conclusions of user satisfaction’s absolute 


measurement of system effectiveness. 


B. JUSTIFICATION SUPPORTED 

Table III contains the studies of Appendix B which have a 
justification for employment of user satisfaction "rial 
supported by the reference literature, 1.e., that the user 
satisfaction tool was used properly according to a cited, 
developed methodology and subsequent conclusions limited to 
the theoretical restrictions of the tool. The judgement 
supported perspective shows that over 40$ of the studies 
employing user satisfaction have incorrectly applied the tool. 
The most common error was to wrongly imply an absolute value 
of system effectiveness from the level of user satisfaction, 
i.e., a successful or unsuccessful system, rather than a 
relative measure, which is a comparison between alternatives 


Or Situations. 
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S TTT satisfaction studies E DEE A 
applied the user satisfaction tool. Those that did limited its 
range of measure to some aspect of relative effectiveness 
rather than employing it as an absolute measure. 

Bight of the 19 are multiple criteria evaluations where 
user Satisfaction was employed as one of several effectiveness 
measures. The multiple criteria approaches in general centered 


around "the relationship between satisfaction and bottom line 


indicators of user and organization performance" (Bailey and 
Pearson, 1983). These bottom line indicators are what this 
E uso considers absolute measures of effectiveness 
(performance). | 


When user satisfaction was employed as a single item 
measure it was most commonly used as a "tool for improving 
system design" (Bailey and Pearson, Lg WD ais 
supplementary or descriptive and thus classified as a relative 
measure by this study. 

Further examination of the studies that use the CWA shows 
that 66$ of them have also incorrectly applied the tool. The 
implication here is that not only have researchers not 
evaluated other methods of measurement as evidenced by the 
CWA, but that they have also not evaluated the user 
satisfaction tool itself and understood its limitations and 


Becerictions in particular those citing Bailey and Pearson. 
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C. REFERENCES TO REJECTION OF ALTERNATIVE MEASURES 

Table V applies a breakdown similar to both CWA and 
Justification Supported with respect to studies which actually 
reference an article which supports the rejection of 
alternative measures to user satisfaction. Common statements 
such as "it is too difficult" or "too costly AAA 
reference, forms another category of conventional wisdom, 
i.e., that alternative methods cannot be used for measuring 
system effectiveness. MIS authors using user satisfaction have 
polarized their view of MIS effectiveness measurement to the 


point that it consists solely of the user satisfaction 


D. REJECTION OF USER SATISFACTION 

Various articles in MIS literature have rejected, whole or 
in part, user satisfaction as a surrogate measure of system 
effectiveness. Hall and Lincoln (1976) "considered the 
possibility of measuring user satisfaction as a surrogate 
measure for information utility ... However it was felt that 
an index of user satisfaction would not have provided senior 
BAC-CAD management with the means to compare the effectiveness 
of information systems with that of other company resource) 
This early rejection of the measure conceptualizes the 
ultimate purpose of any absolute evaluation measure, which is 
an overall assessment. The value of ' which is either for 
judging the success or failure of the system or as a common 


measure to compare one system to another. In either case the 
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value of the measure is complete in itself, i.e., it can stand 
alone and convey valuable information, for justifying company 
resources or comparing completely separate information 
systems. 

The other use to which an evaluation measure is put is to 
compare the relative value within a single item context, i.e., 
better/worse rather than successful/failure. Here again, 
@erei1cS have found fault with user satisfaction on both 
theoretical and methodological grounds. Many reject it saying 
"single item" measures are inadequate (Larker and Lessig, 
NX Hamilton. and Chervany, 1981; Barki and Huff, 1985). 
Pe@ictonally, others (Matlin, 1979; Markus and Robey, 1983; 
and Jarvenpaa et al., 1985) argue that user satisfaction 
measures often lack theory and that the results are subject to 
influence by other factors within the system and therefore it 
is an inconsistent measure and a meaningless tool for 


evaluation. 
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V. CONCLUSIONS 


The evolution of user satisfaction as a surrogate measure 
for IS effectiveness suffers from two deficiences. The first 
is, that in 45 percent of the studies reviewed, it was the 
only method considered, which this study has labled the 
conventional wisdom assertion (CWA). The second stems in part 
from the CWA, that in over 65 percent of the studies the tool 
was improperly applied. 

The MIS community's desire for credibility as a research 
field has led in part to the CWA. A credible user satisfaction 
measurement tool was lacking before that introduced by Pearson 
(1977) and developed by Bailey and Pearson (1983). This prior 
void and the lack of empirical studies in MIS research (Alavi 
et al., 1989) contributed in large part to a blind acceptance 
of the user satisfaction tool as a surrogate measure for 
system effectiveness by those making the conventional wisdom 
assertion. Many in the MIS community grasped the Bailey and 
Pearson tool without regard to its foundation, often 
improperly applying it. By failing to question the newly 
developed tool and even failing to understand its limitations, 
those making the CWA have done a disservice to the MIS 
community and MIS as a whole. Citing Keen (1980) Alavi et al. 
(1989) in their review of MIS research say "Even emerging 


fields such as MIS have a sense of history; those who Cue 
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themselves off from curiosity and reflection are likely to 
upropbhy." 

Additionally, the MIS community's need for empirical 
mesearch (Alavi et al., 1989) facilitated the acceptance of 
user satisfaction aS an empirical tool. There were no 
instances of the CWA in MIS studies prior to Pearson’s 1977 
dissertation and by Bailey and Pearson’s 1983 study 57 percent 
of those that employed user satisfaction made the CWA. This 
highlights the blind acceptance of it by many researchers in 
mae MIS community. 

uu cscecond problem is that of improper application. 
Basically the user satisfaction tool, as developed by Pearson 
and modified by others (Bailey and Pearson, 1983; Ives et al., 
1983) is intended to measure satisfaction as an indicator of 
the relative effectiveness of an information system by means 
Seeevihis/her reaction to the [user satisfaction] factor 
relative to the perceived information requirements” (Bailey 
EmdwPearson, 1983). If the user satisfaction value is higher 
for one case than another the higher valued system will be 
more effective than a lower valued one, whether it is the same 
System which was or will be modified or a totally sepatate 
one. It is not able to evaluate the absolute effectiveness of 
an information system. That would require a comparison between 
inputs and outputs of the system(s). As Marsden and Pingrey 


fees) point out "the key end user, the firm, will be 
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‘satisfied’ when its investment decision, that OA 
information system, contributes to the success of tho 
Therefore, if properly applied employing user satisfaction 
as a surrogate measure for system effectiveness may indicate 
"nore" or "less" effective systems. However, 40 percent of all 
user satisfaction studies reviewed and 57 percent of those 
which were conducted in 1983 and later (Bailey and Pearson) 
improperly applied the tool. This improper application is due 
to its employment as an absolute measure of system 
effectiveness. Combined with the conventional wisdom 
assertion, the implication is that MIS research using user 
satisfaction is on uncertain ground. Not only Ains 
conventional wisdom assertion made by many authors (tablel) 
not justified but when employed user satisfaction is 
improperly applied nearly half the time. The researchers have 
not only accepted the surrogate unquestioningly but they have 
also not bothered to understand and correctly apply it. This 
has a negative effect on the MIS community which is seeking 
credibility as a scientific research field (Alavi et al., 
1989) and for the justification of the financial investmeme. 
being made in information systems (Fortune, 26 May 1990). 
The intent of this study was not to determine the 
scientific validity of user satisfaction as a surto mH 
measure of effectiveness. It was however, intended to shed 
some light on its evolution and use in MIS ihiterictn g 


Examining studies measuring system effectiveness it was 
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determined not only that methods other than user satisfaction 
are "alive and well," thus dispelling the conventional wisdom 
assertion, but also that user satisfaction may be the 


"hobgoblin" of effectiveness measurement (Soergel, 1976). 
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VI. RECOMMENDATIONS 


The role of MIS research is to increase our understanding 
of information systems, thereby allowing prediction of 
outcomes and consequences of the various factors influencing 
an information system. To do so researchers must develop 
theories wien which to guide research and research 
methodologies (Alavi et al., 1989) which provide a systematic 
investigation of these factors. The focus of this study was on 
the measurement of effectiveness of information systems. In 
particular, the evolution -— use of user satisfaction as a 
surrogate measure of effectiveness. At issue was the 
employment of user satisfaction, an opinion, as a measure of 
effectiveness, an action. 

The articles read for this study and the subsequent 
findings suggest several things regarding the measurement of 
effectiveness of information systems. To begin with, the MIS 
community must define a standard framework of information 
system effectiveness. The wide variety of effectiveness 
definitions encountered made it difficult to assess 
validity of any given measure. Additionally, a "systematic 
investigation" is difficult in that findings "arem M 
replicated and verified, and methodologies are not advanced by 


refinement. 
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Next, theories should be developed and tested. The fact 
that almost half of the user satisfaction studies applied the 
CWA and that 40% improperly applied the tool indicates that 
mae field is not thinking about the theory behind the 
relationship. The difficulty that the academic field of 
management science has found linking satisfaction with 
performance should draw INES question an analogous 
relationship a gl MIS between user Satisfaction and 
effectiveness. 

Additionally, social scientists have questioned the 
validity of the survey as a research method and in particular, 
as an accurate measure of behavior. Less than a quarter of the 
user satisfaction studies did any validity checks and only one 
questioned the validity of user satisfaction as a surrogate 
for effectiveness by testing for other possibilities. Instead 
the validity checks merely sought to find a correlation 
between user satisfaction and another attitudinal variable. 

The ubiquitous statements in the literature that 
iitangioles cannot be accurately measured, are too difficult, 
Meo time consuming or too costly to obtain, often without 
reference suggest a general attitude exists in the MIS 
community regarding effectiveness measurement. Combined with 
the fact that the MIS community seems to have embraced the use 
of a subjective surrogate for the same ends, it makes one 
wonder about the motivation or competence behind the research. 


Are we further extending our knowledge of information systems 
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or, is it an easy way to publish and gain tenure? Was the 
research strategy selected the best choice or the easiest? 
The MIS community needs to provide valuable informo ara 
concerning information systems to the user community so that 
they may invest their capital resources more effectively in 
information systems. As Marsden and Pingrey (1988) point out 
"the key end user, the firm, will be ‘satisfied’ when its 
investment decision, that of the information system, 


Contributes tomeme success of thes fice. 
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APPENDIX D. ANNOTATED DESCRIPTION OF USER SATISFACTION 
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Barki, H. "Determinants of User Satisfaction Judgements in 
Information Systems," Proceedings of the Twenty--third Annual 
Hawaii Conference on System Sciences, Vol. VII, (1990), pp. 
408-417. 


Attribution, expectation, disconfirmation and performance 
is determinant of user satisfaction judgements. 


"performance is much more dominate" 


CWA: "In information systems (IS) research, user 

p. 408 satisfaction (or user information satisfactori 
considered to be a key construct for assessing 
system performance." 


JS: "User satisfaction is also used by IS professionals 
W 
p. 408 evaluating the performance ee specific 


information system applications or the overall 
information system in an organization." 


"an implied equivalence between user evaluations 
of system performance and their feelings of 
satisfaction...If such an equivalence exists, then 
measuring satisfaction as a surrogate for system 
performance or effectiveness can be meaningful." 


RR Alt: None 
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IT IT “Examination of Factors. Contributing to the 
Success of the Implementation of Computer-Aided Software 
Engineering Technology," Proceedings of the Twenty-third 
Annual Hawaii Conference on System Sciences, Vol. VI, (1990), 
pe. 418-427. 


"A proposed model consisting of variables that potentially 
influence the development and maintenance processes is 
evaluated in respect to several performance measures." 


CWA : No 

JS: "Satisfaction with the system has been used as a 

p. 421 surrogate for IS effectiveness (Mahmood 1985- 
igo) 

RR Alt: Nae Mec. 


Bemrath, D.W., and O.P. Mignen, "What is Being Done to 
Measure User Satisfaction with EDP/MIS," Information and 
management, Vol. 19, (1990), pp. 7-19. 


If and how organizations are measuring user 
satisfaction what do they perceive as benefits of 
cong So, 


not many organization appear to do it (measure user 
satisfaction) 
those that do value it substantially 


CWA: "There is substantial literature that suggests it 
p. 7 is important to measure user satisfaction with 
EDP/MIS products and services." 


JS: uU cto try tosimprove the malit yas: 
p. 16 (2) to increase the degree... 

(3) to have better relations with users... 

(4) to determine the level of sat. of users... 
RR Alt: None 
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Hufnagel, E.M., "User Satisfaction - Are We Really Measuring 
System Effectiveness?" Proceedings of the Twenty-Third 
Annual Hawail Conference on System Sciences, Vol. VI, 

(1990), pp. 459-4408 


Examines whether "user perception regarding an IS 
provide an accUrate Measure of SyStemmequaliu ys 


"The pattern of causal reasoning observed here raise 
serious questions about the validity of employing user 
Sat. ratings as measures of system effectiveness." 


CWA: "One of the most widely used methods of 
pou evaluating system effectiveness is the user 
satisfaction survey." 


JS: "Sure the hypothesis being tested (H3) is 

p. 441 concerned only with identifying the differences 
in the affective response patterns of two groups 
of performance (as opposed to arriving at a 
"true" measure of value)." 


RR Alt: None 


Baroudi, J.J., and W.J. Orlikowski, "A Short Form Measure of 
User Information Satisfaction: A Psychometric Evaluation and 
Notes on Use," Journal of Management Information Systems, 
Vol. 4, No... 4, Sp names), TET ER 


Develops short form measure of user satisfaction 
from B/P 


CWA : "The measurement of how satisfied a user is with 

pe 44 his or her information system (user into 
satisfied or unsatisfied) has become a pervasive 
measure of the success or effectiveness of an 
information system." 


JS: "has become a pervasive measure of the success or 
p 44-45 effectiveness of an information system" 
RR Alt: None 
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H NT and 6. Torxzaden, “The Measurement of End User 
Computing Satisfaction," Management Information Systems 
EUSrterly, (June 1988), pp. 259-274. 


instrument for measuring end user satisfaction 


CWA: No 
JS: MENA user Computing satisfaction (BUESs) sta 
p. 259 potentially measurable surrogate for utility in 


decis Ton making.” 


RR Alt: “decision. analysis approach 1s generally. not 
E. 259 feasible (Gallagher, 1974, Nolan and Seward, 
Dow 


Evans, et al., "An Instrument for Measuring Effectiveness of 
Information Systems," Computers and Engineering, Vol. 14, 
IDEs 30:980), pp. 227-2396. 


Multi Criteria approach to understand and measure 
information systems 
Int ormatron flows 
user needs 
Meal ETH ETT 
user satisfaction 
alone and in combination are useful for improving 
information systems 


CWA: No 


JS: "After identifying dissatisfied individuals on 

p. 234-235 the Pearson NSI matrix, studying results of the 
Pintore rer ING vaGuals irom Ene. REODOSK 
Methodology can help to show where improvements 
can be made in specific sources of information to 
increase the satisfaction of these individuals." 


p. 228 "alone and in combination are useful for 
improving information systems" 


RR Alt: NA 


6l 


Gallupe, R.B., and G. DeSanctis, "Computer-Based TEI 
Group Problem Finding: An Experimental investigation 
Management Information System Quarterly, Vol. 12, No. 2, 
(June 19898), pp. 2E 


Examines Group DSS en group discussion quality aa 
individual perceptions 


"confidence and satisfaction ... lower in GDSS supp 
groups than in Memecupported Greeupce 


CWA: NA 

JS: "dependent variables were the discussion 

p. 284 performances and perceived reactions 

RR Alt: NA 

Gurmaraes, T., and Y.P. Gupta, "Measuring Top Management 
Satisfaction with the MIS Department," Omega International 


Journal of Management Science, Vol. 16, No. 1, (1988), pp. 
17-24. 


"Questionnaire to measure top management’s satisfaction 
with their MIS department" 


CWA: "researchers have traditionally used measures of 
ES user information satisfaction instead" 

JS: "How should one assess overall MIS department 

po iS performance?" 

RR Alt: "Many of the variables involved in assessing an 
pus MIS department's importance are 


intangible...Because of the difficulty of 
evaluating the economics of information systems." 
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Tait, P., and I. Vessey, "The Effect of User Involvement on 
System Success: A Contingency Approach," Management 
Information Systems Quarterly, (March 1988), pp. 91-108. 


"Path analysis was used to investigate both the direct 
effects of the contingent variables on system success 
and the effect of user involvement." 


CWA : "assessing the satisfaction of the user with the 

mo. 95-96 information provided by the system is probably 
the most common approach to measuring system 
Success “54 - User, Inrormacron satisfaction ls the 
preferred indicator of system success" 


JS: "information satisfaction is the preferred 

pp. 95-96 indicator of system success and is the measure 
used in this study to define successful systems 
development” 


RR Alt: None 


Miller, J., and B.A. Doyle, "Measuring the Effectiveness of 
Computer-Based Information System in the Financial Sector," 
Management Information Systems Quarterly, Vol. 11, No. 1, 
March 1987), pp. 107-124. 


instrument for measuring information system’s 


effectiveness 
CWA: "The instrument used in this research consists of 
p. 108 38 items, 24 of which derive from Pearson, 12 


from Alloway." 
JS: "the effectiveness of the overall information 
=. 107 system’s effort ... is evaluated using an 
instrument based on previous work" 


RR Alt: No 
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Aldag, R.J., and D.J. Power, "An Empirical Assessmen Eiei 
Computer Assisted Decision Analysis," Decision Sciences, 
Vol. 17, No. E IT op. suze 


lab study D HIE eriteria 


CWA: NA 

JS: "Individuals ... will exhibit more confidence mim 
p. BEES and satrtsfascebEOoH with? 

RR Alt: NA 


Baroudi, J.J., M.H. Olson and B. Ives, “An Empirica eee 
of the Impact of User Involvement on System Usage and 
Information Satisfaction," Communications of the ACM E TE 
29, NO. 3/4. (Mame co T2 M 


"that user involvement leads to system usage and/or 
informations a e Sra Ion. 


CWA: "user satisfaction. Generally these constructs 
¡AS are assumed to be indirect indicators of improved 
decision making performance" 


JS: See CWA 


RR Alt: None 


Franz; C.R., and D. Robey, “Organizational Context eee 
Involvement, and the Usefulness of Information Systems," 
Decision Sciences, Vol. 17, No. 3, (Summer 19860), PO TEE 
Sons 


user involvement is positively related to user’s 
perceptions of system usefulness 


CWA: cites Bailey and Pearson (1983) and Ives, Olson 
pom 5 o SN 199.) 

JS: "related positively to user’s perceptions" 

pe 79 

RR Alt: None 
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Manz, C.R., D. Robey, and R.R. Koeblitz, “User Response to 
an Online Information System: A Field Experiment," 
Management Information Systems Quarterly, (March 1986), pp. 
A az. 


"impact of information systems on user attitudes and 
perceorrons of job characteristics" 


CWA: "There seems to be a general consensus that 
p. 32 favorable attitudes are associated with 
successful implementation." 


JS: LUN C uSqgmarprcart.dmpas 
p. 32 Hee cc snonigsesnpeum5oacr" 
RR Alt: None 
Martin, M.P. and J.E. Trumbley, "Measuring Performance of 
Automated Systems," Journal of Systems Management, (February 
Io), pp. 7-17. 

"measuring success of automated systems ... systematic 


structure for measuring" 


CWA: No 

JS: "Kucher’s key concept here is that of comparison" 
p. 14 

RR Alt: NA MC 


DS 


Barki, H., and S.L. Huff, "Change, Attitude to Chang Seo mii 
Decision Support System Success, IN or ra Oan 
Management, Vol. 9, (TO JPET pp. T 


relationships between the extent of changes caused 
by the systems to users’ work environments and the 
users’ attitudes towards work related changes 


concludes DSS users are more satisfied with and use 
more extensively those DSS that bring change than 
those that do mee 


CWA: "To measure system success, researchers have 
p. 4202 mainly relied on either of two proxies: user 
satisfaction or system use" 


JS: "The present study was conducted to develop 
I empirical evidence concerning ... DSS success" 


"User satisfaction is a more appropriate proxy 
[for measuring system success]" 


RR Alt: Yes 
Pa 262 
Hogue, J.T., and H.J. Watson, "An Examination of Decisien. 


Makers’ Utilization of Decision Support System Output 
Information & Management, Vol. 8, No. 4, (August 1985), pp. 
T 

Examines DSS contribution to decision making process 


Dss users are highly satisfied with their systems 


CWA: No 

JS: " .. the evaluation of uSer satisfaction, ee meee 
pou measurement of the value or success of the DSS" 
RR Alt: NA 
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B Ia a c A., "Alternative Measures of System 
Effectiveness: Associations and Implications," Management 
Information Systems Quarterly, Vol. 9, No. 3, (September 
ies), pp. 243-253. 


Reexamines the relationship between user perceived 
effectiveness measures (user Satisfaction) and 
behavioral measures (use) 


oc as live association foumd,- contrary to much of 
existing MIS research 


CWA: "While acknowledging the importance of economic 

p. 243 analysis of MIS value, researchers responded to 
the shifting emphasis from efficiency to user 
effectiveness by focusing either on MIS usage or 
user perceived effectiveness [satisfaction]. Much 
of the MIS literature of late uses one of the 


ona." 

JS: "Measuring and Improving Information System 

p. 243 Effectiveness/Productivity ... Approaches that 
have been advocated include ... user 


Sans rack Lon’ 


RR Alt: Yes 


Baily, J.E., and S.W. Pearson, "Development of A Tool for 
Measuring and Analyzing Computer User Satisfaction," 
management Science, Vol. 29, No. 5, (May 1983), pp. 530-545. 


aol for measuring and analyzing computer user 


satisfaction 
CWA: "It is well recognized that productivity : 
ee 530 FUrtnercr it As- argued. harr utilization- -1S directly 
connected too the user communities’ sense of 
SETE 
JS: "it has been argued that user satisfaction is 
p- 530 correlated to information system utilization and 


System success" 


RR Alt: None 
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Ives, B., M.H. Olson, and J.J. Baroudi, Phe. Ve aI 
User Information Satisfaction," Comm of ACM, Vol. 260m 
10, (October 1983), poet doen. . 


Produces a short form measure of user satisfaction 


CWA: "Satisfaction of users with their information 

p. e Systems is a potentially measurable, and 
generally acceptable, surrogate for utility in 
decision making." 


JS: "MIS provides a meaningful ‘surrogate’ for the 

SL S:5 critical but unmeasurable result of an 
information system, namely, changes in 
organizational effectiveness." 


RR Alt: Yes 
¡O 
King, W..R., and B.J. Epstein, "Assessing Informations" 


Value: An Experimental Study," Decision Sciences, Vol. 14, 
No." 1, (January 9950 Sp. cq-4d5e 


manager assessments of the importance of a Hubritis 
are used to assess information system value. 


CWA: No 
JS: "We want to determine whether practicing 
p. 34 managers’ overall evaluations of ‘total systems’ 


are consistent with those that would be derived 
from linear models that incorporate the managers’ 
evaluations in terms of information attributes." 


RR Alt: NA 
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Mahmood, M.A., "Development and Validation of a Tool for 
Measuring and Analyzing Decision Support Systems User 
BT pp. 171-173. 


Meer Sabistaction is a measure for DSS 


CWA: "User Satisfaction is a good measure of system 

Ep. 171 effectiveness. Furthermore, it is argued that use 
of a system is directly related to the user 
communities’ sense of satisfaction. Measuring and 
analyzing user satisfaction has been of great 
interest ever since the dawn of the computer 


era." 
US: "user satisfaction is a good measure of system 
i. 171 effectiveness" 
RR Alt: None 


Epstein, B.J., and W.R. King, "An Experimental Study of the 
Euer Information," Omega, Vol. 10, No. 3, (1982), pp. 
249-258. 


multidimensional attribute approach to assess the 
value of information 


CWA: No 
JS: "predictions of total system evaluations were 
E 255 made on the basis of self-explicated attribute 


levels and weightings. These predictions were 
then compared with the actual overall system 
assessments that were made by the respondents." 


RR Alt: None 
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Ginzberg, M.J., "Finding an Adequate Measure of OR/MS 
Effectiveness," Interfaces, Vol. 8, No. 4, (Augusto EM 
Ppa OO ZA 


user expectations as predictors of project success 
or fallure 


CWA: "The method for measuring ... success . BENE 

pee 4 on literature has suggested both attitudinal (eg. 
user satisfaction) ..." 

JS: "There are basis in the literature on MIS 

pod implementation and in other fields to support the 


contention that expectations formed early in an 
MIS development project should be related to the 
eventual success or failure of that project." 


RR Alt: None 


Hamilton, S., and N.L. Chervany, "Evaluating Informatie 
System Effectiveness - Part I: Comparing Evaluation 
Approaches," Management Information Systems Quarterly, 
(September 1981), pe. 55-69. 


compares evaluator (user, MIS development, internal 
audit, and top management personnel) viewpoints on 
system effectiveness 


CWA: "User satisfaction with the MIS ... provides the 
li 5) most useful assessment of system effectiveness. 
Substantial evidence supports the validity n 


JS: "the use of a standardized validated instrument 
p. 84 to measure system success" 
RR Alt: None 
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Larcker, D.F., and P.V. Lessig, "Perceived Usefulness of 
Information: A Psychometric Examination," Decision Sciences, 
Perel, No. 1, (January 1980), pp. 121-134. 


Instrument for measuring perceived usefulness 


CWA: D Ene abus pvo MES designers to specitky 

ie, 121 information that is perceived as useful by 
managers is instrumental to the success (or 
failure) of the system" 


JS: (see CWA above) 

p. 121 

RR Alt: None 

E 25. "Experiences Measuring User Satisfaction," 


Proceedings of the Computer Measurement Group of ACM, 
Dallas, (December 1979), pp. 59-66. 


Employs Pearson’s user satisfaction tool to 
Supplement information on FEDSIM review of data 
processing installations 


CWA: NA 

JS: to supplement information ..." 

mE. 59 

RR Alt: NA 

Neumann, S., and E. Segev, "A Case Study of User Evaluation 


of Information Characteristics for Systems Improvement," 
Information & Management, Vol. 2, (1979), pp. 271-278. 


Questionnaire used to evaluate information system 


CWA: NO 

E. 35 

JS: "ra addition to ene relative contreburdgemu 
E. 35 I BermMatl 1 On, SYSECMS gles.” 

RR Alt: No 


gol 


Robey, D., "User Attitudes and Management Information System 
Use," Academy of Management Journal, Vol. 22, No. 3, 
(September 1979) Pron 


attitudes are positively related to use 


CWA : No 

JS: "The primary objective of this research was to 

De oe relate the objective measures of system use to 
attitudes." 

pra or "Strong positive relationships have been 


demonstrated between specific user attitudes and 
actual use of an MIS. It does not necessarily 
follow, however, that attitudes cause behavior." 


RR Alt: None 


King, W.R. and J.I. Rodriguez, "Evaluating Manageme mi 
Information Systems," Management Information Systems 
Quarterly, Vol. 2, No. 3, (September 19/8) m PA: M 


multiple criteria framework for evaluating MIS 


CWA: No 
JS: "These form broad areas of value assessments - 
pe 45 attitudes, value perceptions, information usage 


and decision performance - may be measured at 
various stages of the design and development 
process to evaluate the MIS." 


RR Alt: NA 
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meas, H.C.; Jr., “Application and 
Implementation/Unsuccessful Implementation: The Case of a 
Computer-Based Order Entry System," Decision Sciences, Vol. 
wrens 1, January 1978), pp. 68-795. 


"Successful implementation would be affected by the quality 
of service and the personal costs and benefits to the users" 


CWA : No 

JS: Pocte r1stdctlion." the rndicQqgtor Ol successful 
p. 72 implementation." 

RR Alt: Yes 

p. 69 


Robey, D., and R.L. Zeller, “Factors Affecting the Success 
End Failure of an Information System for Product Quality," 
i cocos Vol. 9, No. 3, (February 1978), pp. 70-75. 


analyze the adoption and rejection of the same 
information system by two similar departments of the 
same company 


CWA: No 
JS: "Responses show the user’s perception of how 
p. 7/1 favorable the system is for each area of concern. 


Questionnaire data can thus provide evidence of 
specific user problems." 

E. 73 "while we cannot say that attitudes were the 
Causes ofi behavior: 


Tue 


Dickson, G.W., J.A. Senn and N.L. Gmervany, “Resear c MERNE 
Management Information Systems: The Minnesota Experiments," 
Management Science, Vol. 23, No. 9, (May 1977), pp. rS TE 


"The Minnesota Experiments ... examine the significance of 
various information system characteristics on decision 
ec egit | 


CWA : NA 


JS: "In addition to discussion quality, measures 
D user evaluation of the information systems used 
were taken ..." 


RR Alt: NA 


Edstrom, A., "User Influence and the Success of MIS Prope eii 
A Contingency Approach," Human Relations, Vol. 30, N PEE 
(1977)7 PP.: D Sar 72 - 


"investigate alternative specifications of the relationship 
between user influence and the success of the management 
information systems" 


Outcome variables: Perceived success (i.e., user 
satısfact ron) 


CWA: "The perceived success of an information 

system 

pus) is, however, not the only possible criterion of 
success and it seems to us to be useful to 
investigate alternative aspects of success." 


JS: "the measures of outcomes [success] are 
Demos perceptual measures M 
RR Alt: None 
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Schewe, C.D., "The Management Information user: An 
Exploratory Behavioral Analysis," Academy of Management 
Matas, Vol. 19, No. 4d, (December 1976), pp. 577-590. 


"Model exploring the relationships between management 
information system users” perceptions of their computer 
system, perceived variables exogenous to the system, 
attitudes and system usage." 


CWA : No 
JS: "According to the model, the beliefs about 
me. 579 characteristics of the MIS are evaluated and 


result in some feeling of favorableness or 
unfavorableness about the information system." 


RR Alt: None 


Gallagher, C.A., "Perceptions of the Value of a Management 
Information System," Academy of Management Journal, Vol. 17, 
Sere, «Maren 1974), pp. 45-55. 


A method of determining the monetary value of an 
existing MIS report is based on user perceptions and 
the semantic differential technique 


CWA : No 

JS: "it seems fair to conclude that the EAB system 
ER Z was well thought of ... as a whole" 

RR Alt: Yes 

pp. 46, 47 


fs 


Lucas, H.C., Jr., "Performance and ThewUse OT an LIII 
System," Management Science, Vol. 21, No. 8, (April 1975a), 
Pp. ONE 


A study of user reactions, the use of information 
Systems and an independent rating of systems 


qua Ey: 
CWA: No 
JS: Hl: "Favorable user attitudes are associated with 
pum favorable user perceptions of systems quality and 


independent ratings of systems quality." 


RR Alt: NA 


Swanson, E.B., "Management Information Systems: Appreciation 
and Involvement," Management Science, Vol. 21, No. 2, 
(October 197/4) eee o 


measurement of involvement and appreciation 


CWA: No 

JS: "An increase in MIS appreciation of an individual 
pe ese will increase his inquiry involvement." 

RR Alt: No 


Powers, R.F., and G.W. Dickson, "MIS Project Management: 
Myths, Opinions and Reality," California Management Review, 
Vol. 157 Noi a lo oe 


Research to identify "the correlates of MIS project 


success” 
CWA: No 
JS: "user satisfaction - attitudes of the managers 
ps 150 receiving project products relative to how well 


their information needs were being satisfied." 


RR Alt: NA 
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APPENDIX E. TABLES OF FINDINGS 


TABLE I. STUDIES MAKING THE CWA 









































Author Year 

Barki A. 20 — 3 
_Conrath/Mignen _ w È K 1 
, Hufnagel | a 90 | | 
, Barooudi/Orlinkowski | . 88 [ 
_Guimaraes/Gupta y 88 ‘ 
Tati/Vessey | E 88 o J 
_Miller/Doyle | | o | 
 Baroudi/Olson/Ives | EE J 
| Franz/Robey | 86 
Franz/Robey/Koeblitz 86 | 
Barki/ Hut & 85 | 
| Srivinason 03 | 
Bailey/Pearson 83 
| Ives/Olson/Baroudi | 83 — 
Mahmood 83 41 
| Ginzberg | 81 | 
, Hamilton/Chervany 81 | 
ME Sees vessig 80 | 
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TABLE TII. STUDIES WITH JUSTIFICATION SUPPORTED 








= ——— — n 






































| AUTHOR YEAR CWA o 
Conrath/Mignen | 90 | Y | 

D um 1 R 
Hufnagel 90 | E | 
Evans, et al. | 88 | NA (MC) | 

CA T | se IAS 

| Aldag/Power | NA (MC) 

| Baroudi/Olson/Ives | 86 | Y l 

| wrtin/trubley —— | e a 

ke 






a 
Srinivason 


| Bailey/Pearson | Y 
| Ives/Olson/Baroudi | 83 | Y 
King/Epstein 83 
Deese 79 | NA (FEDSIM) 
| Nueman/Segev mo | | 
| Robey X9 | 


King/Rodriguez —n" | 
[Robey/Zellar — | 78 | N 
Schewe 
, Swanson eo 


| Powers/Dickson 





TABLE IV. BREAKDOWN OF JUSTIFICATION SUPPORTED 


| All yrs. 24/40 Ze 57% 






+CWA |. 8/18 44$ 


| 
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TABLE V. STUDIES CITING A REFERENCE REJECTING ALTERNATIVES 
























































(RR Alt) 
AUTHOR YEAR E CWA 
_Doll/Torkzadeh 6 88 i N 
Barki/Huff , 05 i Y 
Srinivason L 85 4 = 
| Ives/Olson/Baroudi , 83 , t 
PER BEEN | 78 X B 
| Gallagher 74 | N 
q O NA : nn 
m Yrs | 6/32 h^ 198 
| TCWA — | SED 9 Xu 
| ' 83+ EE [ 23% 
77+ 3/18 17% 





T9 























TABLE VII. 








"m a 





T 


Powers/Dickson 








NAMES, DEFINITION AND CONSTRUCT OF USER 
SATISFACTION VARIABLES 


————— 








VARIABLE: DEFINITION 
CONSTANT 





user satisfaction: attitudes of managers 








L ae | not descrioed | 
N — o. _AA —_ _  __ eee eee, 
| | 
2. Swanson appreciation: users beliefs about relative value | 
| ' 14 | not described 
|| 
| 3. eas | user attitudes: not defined | 
| ' 14 some scale not described 
. 4. Gallagher | user perceptions: not defined 
' 714 | semantic differential 
| 5. Schewe user perceptions, attitudes: not defined 
| "de not described | 
| 6. JEGStrom perceived success: not defined 
| er complete success - complete failure 
| 7. Dukson/Senn/Chervany user evaluation: not defined | 
l em, not described 
8. Robey/Zeller attitudes: not defined | 
| S Shultz-Slevin | 
i| 
| : e | 
| aS DUC user perceptions: modified JDI | 
| ’ 78 Shultz-Slevin 
| 10. King/Roariquez value perceptions: "direct assessment" of MIS | 
l) TS Shultz-Slevin 
| 11. Robey user attitudes: not defined 
A79 Shultz-Slevin 
12. Neumann/Segev user evaluation: accuracy, content, frequency, recency | 
e questionnaire | 
| 13. Deese ser —Ssacisfacti oni | 
| ' 19 Pearson method | 
| 
t , a | 
14. Larkin/Lessig perceived useableness/perceived importance | 
| 80 questionnaire - 7 pt scale 
| 19. Hamilton/Chervany üser satisfaction.: 
i oul | Pearson method 
| 
16. Ginzberg user satisfaction 
"B1 Shultz-Slevin 
17. Epstein/King satisfaction/value perception: 10 attributes 
' 83 guestionnarie, 5 pt scale 
18. Mahmood user satisfaction: * | 
"Hd Pearson method | 
| 19. King/Epstein satisfaction/value perception: 10 attributes | 
283 questionnaire, 2 pt scale 
20. Ives/Olson/Baroudi user satisfaction: * 
ao Pearson, Bailey/Pearson 
21. Bailey/Pearson user satisfaction: semantic differential factors 39 
"88 Pearson method 
| 22. Srinivasen user satisfaction: * 
235 Bailey/Pearson 
23. Hogue/Watson perceived value: satisfaction/diissatisfaction 
BS questionnaire 7 pt scale 
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So 


VARIABLE: DEFINITION 








STUDY 
YEAR CONSTANT 

pu ————— PH € <A A A ——À ——— > —— Å- -— 
weed. Barer hare USC lesec. sac ION 

SS Pearson Me ned 

n x 
M25. Martin/TrhümblMy JSer accommodation: "ruie ef satisfiting"™ 
i 8/6 questionnaire - Lixert scales 
= A eee rae Pees ee e = 

26. Pranz/Robey/XoebiR*z user attitudes/percept ions 

"86 Schultz-Slevin 
|j 27. Franz/Robey perceivea usefulness: user satisfaction * 

"HG Bailey and Pearson 


—— 
| 28. Baroudi/Olson/Ives 


| 29. Aldag/Power 
"DG 





SITE T DOV 1.6 
DT 


31. Tart/Vessey 
' 88 


| 32. Gurmaraes/Gupta 
| ' 88 


33. Gallupe/DeSanctis 
| ' 88 


EJ Evans, et. al 
' 88 


Bo Doll /Torkzadeh 
| ' 88 


36. Baroudi/Orlikowski 


EB 





37. Hufnagel 
30 


38. Conrath/Mignen 
' 90 


39. Burkhard 


90 








user satisfaction: * 


* B6 Bailey and Pearson / 
f 


user perceptions: multipie item 
Likert scale 


user Satistaction: * 
Bailey and Pearson 


user satisfaction: * 
Bailey and Pearson 


user satisfaction: * 
Bailey and Pearson 


perceived reactions, satisfaction 
questionnaire, Likert scale 


user satisfaction: * 
Pearson Method 


user satisfaction: * | 
Bailey and Pearson 





user satisfaction: short form 13 factors * 
Bailey and Pearson 


user satisfaction: * 
Bailey and Pearson 


user satisfaction: * 
Bailey and Pearson 


User satisfaction: * 
Bailey and Pearson short form 


user satisfaction: * | 
Bailey and Pearson 


* see "USER SATISFACTION" in Section III, METHODOLOGY 


oh 





e — 


TABLE VIII. STUDIES WITH VALIDITY/RELIABILITY CHECKS 














Hufnagel Io | 
, Baroudi/Orlikowski | ^'88 
|Doll/DonkEzagcH ‘88 
 Gurmaraes/Gupta | R 
Mart dese cy | ' 88 
Miller ce | oe 
| Bailey/Pearson ' 83 | 


ives) Olsen Sarcudi 


| Mahmood 


J Larcker/Lessey 
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